Open peer review urgently requires evidence: A call to action.

Open Peer Review is gaining prominence in attention and use, but to responsibly open up peer review, there is an urgent need for additional evidence. Here, we propose a preliminary research agenda and issue a call to action.

Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review.

BACKGROUND: The emergence of systems based on large language models (LLMs) such as OpenAI's ChatGPT has created a range of discussions in scholarly circles. Since LLMs generate grammatically correct and mostly relevant (yet sometimes outright wrong, irrelevant or biased) outputs in response to provided prompts, using them in various writing tasks including writing peer review reports could result in improved productivity. Given the significance of peer reviews in the existing scholarly publication landscape, exploring challenges and opportunities of using LLMs in peer review seems urgent. After the generation of the first scholarly outputs with LLMs, we anticipate that peer review reports too would be generated with the help of these systems. However, there are currently no guidelines on how these systems should be used in review tasks. METHODS: To investigate the potential impact of using LLMs on the peer review process, we used five core themes within discussions about peer review suggested by Tennant and Ross-Hellauer. These include 1) reviewers' role, 2) editors' role, 3) functions and quality of peer reviews, 4) reproducibility, and 5) the social and epistemic functions of peer reviews. We provide a small-scale exploration of ChatGPT's performance regarding identified issues. RESULTS: LLMs have the potential to substantially alter the role of both peer reviewers and editors. Through supporting both actors in efficiently writing constructive reports or decision letters, LLMs can facilitate higher quality review and address issues of review shortage. However, the fundamental opacity of LLMs' training data, inner workings, data handling, and development processes raise concerns about potential biases, confidentiality and the reproducibility of review reports. Additionally, as editorial work has a prominent function in defining and shaping epistemic communities, as well as negotiating normative frameworks within such communities, partly outsourcing this work to LLMs might have unforeseen consequences for social and epistemic relations within academia. Regarding performance, we identified major enhancements in a short period and expect LLMs to continue developing. CONCLUSIONS: We believe that LLMs are likely to have a profound impact on academia and scholarly communication. While potentially beneficial to the scholarly communication system, many uncertainties remain and their use is not without risks. In particular, concerns about the amplification of existing biases and inequalities in access to appropriate infrastructure warrant further attention. For the moment, we recommend that if LLMs are used to write scholarly reviews and decision letters, reviewers and editors should disclose their use and accept full responsibility for data security and confidentiality, and their reports' accuracy, tone, reasoning and originality.

Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other Large Language Models in scholarly peer review.

Background: The emergence of systems based on large language models (LLMs) such as OpenAI's ChatGPT has created a range of discussions in scholarly circles. Since LLMs generate grammatically correct and mostly relevant (yet sometimes outright wrong, irrelevant or biased) outputs in response to provided prompts, using them in various writing tasks including writing peer review reports could result in improved productivity. Given the significance of peer reviews in the existing scholarly publication landscape, exploring challenges and opportunities of using LLMs in peer review seems urgent. After the generation of the first scholarly outputs with LLMs, we anticipate that peer review reports too would be generated with the help of these systems. However, there are currently no guidelines on how these systems should be used in review tasks. Methods: To investigate the potential impact of using LLMs on the peer review process, we used five core themes within discussions about peer review suggested by Tennant and Ross-Hellauer. These include 1) reviewers' role, 2) editors' role, 3) functions and quality of peer reviews, 4) reproducibility, and 5) the social and epistemic functions of peer reviews. We provide a small-scale exploration of ChatGPT's performance regarding identified issues. Results: LLMs have the potential to substantially alter the role of both peer reviewers and editors. Through supporting both actors in efficiently writing constructive reports or decision letters, LLMs can facilitate higher quality review and address issues of review shortage. However, the fundamental opacity of LLMs' inner workings and development, raise questions and concerns about potential biases and the reliability of review reports. Additionally, as editorial work has a prominent function in defining and shaping epistemic communities, as well as negotiating normative frameworks within such communities, partly outsourcing this work to LLMs might have unforeseen consequences for social and epistemic relations within academia. Regarding performance, we identified major enhancements in only a few weeks (between December 2022 and January 2023) and expect ChatGPT to continue improving. Conclusions: We believe that LLMs are likely to have a profound impact on academia and scholarly communication. While they have the potential to address several current issues within the scholarly communication system, many uncertainties remain and their use is not without risks. In particular, concerns about the amplification of existing biases and inequalities in access to appropriate infrastructure warrant further attention. For the moment, we recommend that if LLMs are used to write scholarly reviews, reviewers should disclose their use and accept full responsibility for their reports' accuracy, tone, reasoning and originality.

Messing with Merton: The intersection between open science practices and Mertonian values.

Although adherence to Mertonian values of science (i.e., communism, universalism, organized skepticism, disinterestedness) is desired and promoted in academia, such adherence can cause friction with the normative structures and practices of Open Science. Mertonian values and Open Science practices aim to improve the conduct and communication of research and are promoted by institutional actors. However, Mertonian values remain mostly idealistic and contextualized in local and disciplinary cultures and Open Science practices rely heavily on third-party resources and technology that are not equally accessible to all parties. Furthermore, although still popular, Mertonian values were developed in a different institutional and political context. In this article, we argue that new normative structures for science need to look beyond nostalgia and consider aspirations and outcomes of Open Science practices. To contribute to such a vision, we explore the intersection of several Open Science practices with Mertonian values to flesh out challenges involved in upholding these values. We demonstrate that this intersection becomes complicated when the interests of numerous groups collide and contrast. Acknowledging and exploring such tensions informs our understanding of researchers' behavior and supports efforts that seek to improve researchers' interactions with other normative structures such as research ethics and integrity frameworks.

Research funders should be more transparent: a plea for open applications.

Transparency is increasingly becoming the new norm and modus operandi of the global research enterprise. In this mini-review, we summarize ongoing initiatives to increase transparency in science and funding in particular. Based on this, we make a plea for the next step in funders' compliance with the principles of Open Science, suggesting the adoption of open applications. Our proposed model includes a plea for the publication of all submitted grant applications; open sharing of review reports, argumentations for funding decisions and project evaluation reports; and the disclosure of reviewers' and decision committee members' identities. In line with previous calls for transparency and the available evidence about these measures' effectiveness, we argue that open applications could lead to more diverse collaboration, recognition of research ideas, fairer procedures for grant allocation, more research on funding practices and increased trust in the funding allocation process.

Designing and implementing a research integrity promotion plan: Recommendations for research funders.

Various stakeholders in science have put research integrity high on their agenda. Among them, research funders are prominently placed to foster research integrity by requiring that the organizations and individual researchers they support make an explicit commitment to research integrity. Moreover, funders need to adopt appropriate research integrity practices themselves. To facilitate this, we recommend that funders develop and implement a Research Integrity Promotion Plan (RIPP). This Consensus View offers a range of examples of how funders are already promoting research integrity, distills 6 core topics that funders should cover in a RIPP, and provides guidelines on how to develop and implement a RIPP. We believe that the 6 core topics we put forward will guide funders towards strengthening research integrity policy in their organization and guide the researchers and research organizations they fund.

Misidentified biomedical resources: Journal guidelines are not a quick fix.

Biomedical researchers routinely use a variety of biological models and resources, such as cultured cell lines, antibodies and laboratory animals. Unfortunately, these resources are not flawless: cell lines can be misidentified; for antibodies, problems with specificity, lot-to-lot consistency and sensitivity are common; and the reliability of animal models is questioned due to poor translation of animal studies to human clinical trials. In some cases, these problems can render the results of a study meaningless. As a response, some journals have implemented guidelines regarding the use and reporting of cell lines, antibodies and laboratory animals. In our study we use a portfolio of existing and newly created datasets to investigate identification and authentication information of cell lines, antibodies and organisms before and after guideline introduction, compared to journals without guidelines. We observed a general improvement of reporting quality over time, which the implementation of guidelines accelerated only in some cases. We therefore conclude that the effectiveness of journal guidelines is likely to be context dependent, affected by factors such as implementation conditions, research community support and monitoring and resource availability. Hence, journal reporting guidelines in themselves are not a quick fix to repair shortcomings in biomedical resource documentation, even though they can be part of the solution.

Towards a Research Agenda for Promoting Responsible Research Practices.

This opinion piece aims to inform future research funding programs on responsible research practices (RRP) based on three specific objectives: (1) to give a sketch of the current international discussion on responsible research practices (RRPs); (2) to give an overview of current initiatives and already obtained results regarding RRP; and (3) to give an overview of potential future needs for research on RRP. In this opinion piece, we have used seven iterative methodological steps (including literature review, ranking, and sorting exercises) to create the proposed research agenda. We identified six main themes that we believe need attention in future research: (1) responsible evaluation of research and researchers, (2) the influence of open science and transparency on RRP, (3) research on responsible mentoring, supervision, and role modeling, (4) the effect of education and training on RRP, (5) checking for reproducibility, and (6) responsible and fair peer review. These themes have in common that they address aspects of research that are mostly on the level of the scientific system, more than on the level of the individual researcher. Some current initiatives are already gathering substantial empirical evidence to start filling these gaps. We believe that with sufficient support from all relevant stakeholders, more progress can be made.

Innovating editorial practices: academic publishers at work.

BACKGROUND: Triggered by a series of controversies and diversifying expectations of editorial practices, several innovative peer review procedures and supporting technologies have been proposed. However, adoption of these new initiatives seems slow. This raises questions about the wider conditions for peer review change and about the considerations that inform decisions to innovate. We set out to study the structure of commercial publishers' editorial process, to reveal how the benefits of peer review innovations are understood, and to describe the considerations that inform the implementation of innovations. METHODS: We carried out field visits to the editorial office of two large academic publishers housing the editorial staff of several hundreds of journals, to study their editorial process, and interviewed editors not affiliated with large publishers. Field notes were transcribed and analysed using coding software. RESULTS: At the publishers we analysed, the decision-making structure seems to show both clear patterns of hierarchy and layering of the different editorial practices. While information about new initiatives circulates widely, their implementation depends on assessment of stakeholder's wishes, impact on reputation, efficiency and implementation costs, with final decisions left to managers at the top of the internal hierarchy. Main tensions arise between commercial and substantial arguments. The editorial process is closely connected to commercial practices of creating business value, and the very specific terms in which business value is understood, such as reputation considerations and the urge to increase efficiency. Journals independent of large commercial publishers tend to have less hierarchically structured processes, report more flexibility to implement innovations, and to a greater extent decouple commercial and editorial perspectives. CONCLUSION: Our study demonstrates that peer review innovations are partly to be understood in light of commercial considerations related to reputation, efficiency and implementations costs. These arguments extend beyond previously studied topics in publishing economics, including publishers' choice for business or publication models and reach into the very heart of the editorial and peer review process.

On the Willingness to Report and the Consequences of Reporting Research Misconduct: The Role of Power Relations.

While attention to research integrity has been growing over the past decades, the processes of signalling and denouncing cases of research misconduct remain largely unstudied. In this article, we develop a theoretically and empirically informed understanding of the causes and consequences of reporting research misconduct in terms of power relations. We study the reporting process based on a multinational survey at eight European universities (N = 1126). Using qualitative data that witnesses of research misconduct or of questionable research practices provided, we aim to examine actors' rationales for reporting and not reporting misconduct, how they report it and the perceived consequences of reporting. In particular we study how research seniority, the temporality of work appointments, and gender could impact the likelihood of cases being reported and of reporting leading to constructive organisational changes. Our findings suggest that these aspects of power relations play a role in the reporting of research misconduct. Our analysis contributes to a better understanding of research misconduct in an academic context. Specifically, we elucidate the processes that affect researchers' ability and willingness to report research misconduct, and the likelihood of universities taking action. Based on our findings, we outline specific propositions that future research can test as well as provide recommendations for policy improvement.

The ghosts of HeLa: How cell line misidentification contaminates the scientific literature.

While problems with cell line misidentification have been known for decades, an unknown number of published papers remains in circulation reporting on the wrong cells without warning or correction. Here we attempt to make a conservative estimate of this 'contaminated' literature. We found 32,755 articles reporting on research with misidentified cells, in turn cited by an estimated half a million other papers. The contamination of the literature is not decreasing over time and is anything but restricted to countries in the periphery of global science. The decades-old and often contentious attempts to stop misidentification of cell lines have proven to be insufficient. The contamination of the literature calls for a fair and reasonable notification system, warning users and readers to interpret these papers with appropriate care.